Computer Vision Assisted Work Tool Recognition and Installation

ABSTRACT

A method for installing a work tool for a machine is provided. The method includes detecting, at an electronic controller unit of a machine, a work tool based upon a first input signal from a sensor coupled to the electronic controller unit. The method includes determining, at the electronic controller unit, a first three-dimensional location of the work tool relative to the machine. The method includes detecting, at the electronic controller unit, an occlusion of the work tool. The method includes determining, at the electronic controller unit, a second three-dimensional location of the work tool upon the detecting of the occlusion based on the first three-dimensional location. The method includes controlling, at the electronic controller unit, a motion of the machine for installing the work tool based upon the second three-dimensional location.

TECHNICAL FIELD

This patent disclosure relates generally to attachment of work tools toa machine, and more particularly, to a method and a system for computervision assisted work tool recognition and installation for a machine.

BACKGROUND

Machines have multiple types of work tools or attachments for differentwork purposes. For example, a wheel loader may use a bucket for movingearth, and may use a fork for picking up pallets. Changing work tools ina safe and quick fashion is one of the basic qualifications of a goodmachine operator. However, it may require a new operator a long time oftraining to master this skill.

A typical challenge encountered by the machine operator during achanging of the work tool is that additional manual operations may berequired for changing the machine's output mode settings. Usually, themachine operator has to select a proper machine output mode for aspecific work tool by manually pressing a button on a control panel.Forgetting to press the button or inadvertently selecting a wrong outputmode may cause the machine to malfunction.

Another challenge for the machine operator is a limited front field ofview of the work tool. The view of the work tool to be installed orattached may be blocked by some mechanical parts on a machine. Forexample, the hydraulic cylinders and mechanical linkages in front of anoperator cab of a wheel loader may block the machine operator's view ofa fork to be installed or attached to the machine.

Yet another challenge faced by the machine operator is the difficulty ofaligning an attachment coupler of the work tool at a distance from themachine to which the work tool attaches. When the work tool to beinstalled is at a distance from the machine, it is difficult for themachine operator to manually accurately align an attachment coupler ofthe work tool with the machine due to the distance.

WO 2014046313 discusses capturing an image of an attachment andrecognizing the attachment using a database. However, conventionalsystems and methods do not address the challenges faced by the machineoperator, for example, when the view of the work tool is blocked.

Accordingly, there is a need to resolve these problems and otherproblems related to conventional methods and systems used for attachingwork tool to machines in order to reduce training time for newoperators, and increase work productivity.

SUMMARY

In one aspect of this disclosure, a method for installing a work toolfor a machine is provided. The method includes detecting, at anelectronic controller unit of a machine, a work tool based upon a firstinput signal from a sensor coupled to the electronic controller unit.The method includes determining, at the electronic controller unit, afirst three-dimensional location of the work tool relative to themachine. The method includes detecting, at the electronic controllerunit, an occlusion of the work tool. The method includes determining, atthe electronic controller unit, a second three-dimensional location ofthe work tool upon the detecting of the occlusion based on the firstthree-dimensional location. The method includes controlling, at theelectronic controller unit, a motion of the machine for installing thework tool based upon the second three-dimensional location.

In another aspect of this disclosure, a work tool installation system isprovided. The work tool installation system includes a machineattachable to a work tool. The machine includes a sensor and anelectronic controller unit coupled to the sensor. The electroniccontroller unit is configured to detect the work tool based upon a firstinput signal from the sensor, determine a first three-dimensionallocation of the work tool relative to the machine, detect an occlusionof the work tool, determine a second three-dimensional location of thework tool, when the occlusion is detected, based on the firstthree-dimensional location, and control a motion of the machine forinstalling the work tool based upon the second three-dimensionallocation.

In yet another aspect of this disclosure, an electronic controller unitof a machine is provided. The electronic controller unit includes amemory and a processor. The memory includes computer executableinstructions for recognizing and installing a work tool to a machine.The processor is coupled to the memory and configured to execute thecomputer executable instructions, the computer executable instructionswhen executed by the processor cause the processor to detect the worktool based upon a first input signal from a sensor, determine a firstthree-dimensional location of the work tool relative to the machine,detect an occlusion of the work tool, determine a secondthree-dimensional location of the work tool, when the occlusion isdetected, based on the first three-dimensional location, and control amotion of the machine for installing the work tool based upon the secondthree-dimensional location.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a work tool installation system, in accordance withan aspect of this disclosure.

FIG. 2 illustrates a view of a work tool from inside an operator cab ofa machine in the work tool installation system of FIG. 1, in accordancewith an aspect of this disclosure.

FIG. 3 illustrates the work tool installation system of FIG. 1 with themachine moved closer to the work tool, in accordance with an aspect ofthis disclosure.

FIG. 4 illustrates a view of the work tool from inside the operator cabwith a view of the work tool partially occluded, in accordance with anaspect of this disclosure.

FIG. 5 illustrates a schematic block diagram of the work toolinstallation system of FIG. 1, in accordance with an aspect of thisdisclosure.

FIG. 6 illustrates a first frame showing a three-dimensional scene ofthe work tool installation system of FIG. 1, in accordance with anaspect of this disclosure.

FIG. 7 illustrates a second frame showing a three-dimensional scene ofthe work tool installation system of FIG. 3, in accordance with anaspect of this disclosure.

FIG. 8 illustrates a flowchart for a method for installing a work toolfor a machine, in accordance with an aspect of this disclosure.

DETAILED DESCRIPTION

Various aspects of this disclosure are related to addressing theproblems in the conventional machines and methods by usingcomputer-vision assisted work tool recognition and installation.

Now referring to the drawings, where like reference numerals refer tolike elements, FIG. 1 illustrates a work tool installation system 100,in accordance with an aspect of this disclosure. The work toolinstallation system 100 includes a machine 102 and a work tool 104 in anexemplary work environment. It will be appreciated that the work toolinstallation system 100 may include a plurality of machines and aplurality of work tools and the machine 102 and the work tool 104illustrated in FIG. 1 are by way of example only and not by way oflimitation. Further, the work tool installation system 100 may includeadditional components, including but not limited to, a base station incommunication with the machine 102, a satellite system in communicationwith the machine 102, an unmanned aerial vehicle in communication withthe machine 102, and the like, to assist recognition and installation ofthe work tool 104 to the machine 102.

The machine 102 may be a movable machine or a stationary machine havingmovable parts. In this respect, the term “movable” may refer to a motionof the machine 102, or a part thereof, along linear Cartesian axes,and/or along angular, cylindrical, or helical coordinates, and/orcombinations thereof. Such motion of the machine 102 may be continuousor discrete in time. For example, the machine 102, and/or a part of themachine 102, may undergo a linear motion, an angular motion or both.Such linear and angular motion may include acceleration, rotation aboutan axis, or both. By way of example only and not by way of limitation,the machine 102 may be an excavator, a paver, a dozer, a skid steerloader (SSL), a multi-terrain loader (MTL), a compact track loader(CTL), a compact wheel loader (CWL), a harvester, a mower, a driller, ahammer-head, a ship, a boat, a locomotive, an automobile, a tractor, orother machine to which the work tool 104 is attachable.

In the example shown in FIG. 1, the machine 102 includes a machinecomponent 108, a sensor 110, an operator cab 112, a chassis 114, tires116, and a hood 118. The machine component 108 is attachable to the worktool 104 at an attachment coupler 106 of the work tool 104. The operatorcab 112 includes, among other components, a steering system 124 to guidethe machine 102 in various spatial directions, and an output device 140.The operator cab 112 may be suitably sized to accommodate a humanoperator. Alternatively, the machine 102 may be controlled remotely froma base station, in which case, the operator cab 112 may be smaller. Thesteering system 124 may be a steering wheel or a joystick, or othercontrol mechanism to guide a motion of the machine 102, or partsthereof. Further, the operator cab 112 may include levers, knobs, dials,displays, alarms, etc. to facilitate operation of the machine 102.

Under the hood 118, the machine 102 includes an electronic controllerunit 126, an inertial measurement unit (IMU) 128, and a machine controlsystem 130. The machine 102 may include other components (e.g., as partof the chassis 114) such as transmission systems, engine(s), motors,power system(s), hydraulic system(s), suspension systems, coolingsystems, fuel systems, exhaust systems, ground engaging tools, anchorsystems, propelling systems, communication systems including antennas,Global Positioning Systems (GPS), and the like (not shown) that arecoupled to the machine control system 130.

By way of example only and not by way of limitation, the machinecomponent 108 may be an excavator arm including hydraulic cylinders andmechanical linkages, although other types of mechanical parts may beutilized to attach the work tool 104 to the machine 102. The mechanicallinkages may include attachment components compatible to mate with theattachment coupler 106. The machine component 108 may be extendable,expandable, contractable, rotatable, translatable radially or axially,or otherwise movable by the machine 102 to couple to the work tool 104.For example, a height and a tilt of the machine component 108 may bevariable to facilitate attachment at the attachment coupler 106. Onceattached to the work tool 104, the machine component 108 may beconfigured to receive requisite power from the machine 102 to performvarious operations (e.g., digging earth) in the exemplary worksite usingthe work tool 104.

In one aspect of this disclosure, the sensor 110 may be a camerapositioned on, inside, or above the operator cab 112. Alternatively, thesensor 110 may be a camera positioned on the machine component 108,e.g., near or at a front end of the machine component 108 closest to thework tool 104, although the sensor 110 may be positioned at otherlocations on the machine 102. By way of example only and not by way oflimitation, the sensor 110 may be a monocular camera, a stereo camera,an infrared camera, an array of one or more types of cameras, anopto-acoustic sensor, a radar, a laser based imaging sensor, or thelike, or combinations thereof, configured to assist recognition,detection, tracking, and installation of the work tool 104.

The work tool 104 is attachable to the machine 102, for example, to alinkage at an end portion of the machine component 108 via theattachment coupler 106. By way of example only and not by limitation,the work tool 104 may be a bucket for moving earth, a fork for liftingpallets, a harvester attachment, a drill head, a hammer head, acompactor head, or any other type of implement attachable to the machine102. In this respect, the machine 102 may be configured to be attachablenot just to one type of the work tool 104, but also to different typesof the work tool 104, as well as to a plurality of work tools at thesame time. Depending on the type of the work tool 104, the machine 102may be configured to operate in an output mode specific to the type ofthe work tool 104. An output mode of the machine 102 is specified byappropriate electrical and mechanical parameters for operation of thework tool 104 when attached to the machine component 108. For example,an output mode for a bucket is different from an output mode of a forkin terms of an output power delivered to the work tool 104. If anincorrect output mode is selected, or if no output mode is selected by amanual operator when the work tool 104 is attached to the machinecomponent 108, the machine 102 may not be able to properly perform, ornot perform, the job for which the machine 102 was deployed. Further,depending on the type of the work tool 104, the attachment coupler 106may be an attachment pin, a latch, a hook, a ball/socket joint, or othertypes of attachment components that make the work tool 104 couplable tothe machine component 108 of the machine 102. In one aspect, the worktool 104 may be stationary. In another aspect, the work tool 104 may bemobile or movable towards the machine 102. For example, another machine(not shown) may be used to push the work tool 104 to match a motion ofthe machine 102 and/or of the machine component 108.

As illustrated in FIG. 1, the work tool 104 is at an initial positionrelative to the machine 102. The initial position may be determined as afirst three-dimensional location 120 of the work tool 104 from thesensor 110. At this initial position, FIG. 2 illustrates a view of thework tool 104 from inside the operator cab 112, in accordance with anaspect of this disclosure. As shown in FIG. 2, a view of the work tool104 is not blocked and the work tool 104 is clearly visible withoutobstruction from the machine component 108, e.g., to an operator in theoperator cab 112.

As the machine 102 moves closer to the work tool 104, as illustrated inFIG. 3, the initial position indicated by the first three-dimensionallocation 120 changes. The work tool 104 102 is now determined to be at asecond three-dimensional location 220 from the machine 102. The machine102 may move towards the work tool 104 continuously or discretely.Regardless of the way the machine 102 moves, the relative position ofthe work tool 104 and the machine 102 is updated.

Referring back to FIG. 1, in one aspect, the attachment coupler 106 isdetermined to be at a third three-dimensional location 122 in theinitial position. Again, as the machine 102 moves, the relative distanceor the relative position between the attachment coupler 106 and themachine 102 changes, using the location of the sensor 110 as areference.

For example, referring to FIG. 3, the attachment coupler 106 is at afourth three-dimensional location 222 from the machine 102. Further, thefirst three-dimensional location 120, the second three-dimensionallocation 220, the third three-dimensional location 122, and the fourththree-dimensional location 222 may be measured between various points onthe machine 102, the machine component 108, the work tool 104, and theattachment coupler 106, and the illustrated positions shown in FIGS. 1and 3 are by way of example only and not by way of limitation.

It will be appreciated that the terms “first”, “second”, “third”, and“fourth” used herein with respect to the initial, intermediate, or finalpositions of the machine 102 and the machine component 108 relative tothe work tool 104 are for differentiating purposes only and not for anyparticular priority in which such relative positions between the machine102, the machine component 108, and the work tool 104 are effectuated.Although illustrated as linear distances (as indicated by respectivestraight lines), the first three-dimensional location 120, the secondthree-dimensional location 220, the third three-dimensional location122, and the fourth three-dimensional location 222, as well as otherintermediate three-dimensional locations for intermediate relativepositions of the machine 102 with respect to the work tool 104, may bevectors expressible in one or more coordinate systems and stored in amemory 508 (shown in FIG. 5) of the electronic controller unit 126.

Referring to FIG. 4, another view of the work tool 104 from inside theoperator cab 112 is illustrated as the machine 102 moves closer to thework tool 104, in accordance with an aspect of this disclosure. The viewof the work tool 104 illustrated in FIG. 4 is occluded during operationof the machine 102, for example, by a presence of the machine component108. In one aspect, such an occlusion of the work tool 104 may result inthe attachment coupler 106 not being visible (fully or partially) to theoperator of the machine 102. In this respect, the term “occlusion”relates to a partial or complete blocking of a view of the work tool 104and/or the attachment coupler 106 due to a presence of a component(e.g., the machine component 108) between the work tool 104 and/or theattachment coupler 106 and the operator or the sensor 110 that views thework tool 104 and/or the attachment coupler 106. It will be appreciatedthat the occlusion of the work tool 104 and/or the attachment coupler106 may be due to the machine component 108, or due to other componentsnot belonging to the machine 102, or both.

In one aspect of this disclosure, the machine control system 130 mayinclude various hydraulic and electrical power systems controlled by theelectronic controller unit 126, based upon output signals from theelectronic controller unit 126 to the machine control system 130. Themachine control system 130 may include or may be coupled to the steeringsystem 124 configured to guide a motion of the machine 102 and/or themachine component 108. In another aspect, the machine control system130, or a part thereof, may be located remote from the machine 102,e.g., in a base station physically separated from the machine 102. Inthis scenario, the machine control system 130 may have a direct orindirect communication link with the electronic controller unit 126 tocontrol the machine 102 for installing the work tool 104.

Referring to FIG. 5, a schematic diagram of the work tool installationsystem 100 with the machine 102 including the electronic controller unit126 is illustrated, in accordance with an aspect of this disclosure. Theelectronic controller unit 126 is coupled to the sensor 110, theinertial measurement unit 128, the machine control system 130, theoutput device 140, and the steering system 124, as well as to othercomponents of the machine 102 (not shown).

In one aspect of this disclosure, the sensor 110 has a field of view 502within which the work tool 104 and/or the attachment coupler 106 fall.During an occlusion of the work tool 104 and/or the attachment coupler106, as discussed, the machine component 108 may fall within the fieldof view 502 of the sensor 110 to partially or fully block a view of thework tool 104 and/or the attachment coupler 106. This may prevent thesensor 110 from obtaining a full image of the work tool 104 and/or theattachment coupler 106. In conventional systems, such occlusion may slowdown or make it difficult to attach the work tool 104 to the machinecomponent 108, and requires manual intervention that is disruptive tothe changing and installation process of the work tool 104. To addressthis issue, the electronic controller unit 126 may continuously receivean input signal 518 from the sensor 110 at an input-output port 504 ofthe electronic controller unit 126. The input signal 518 may includeinformation regarding a current or an updated three-dimensional locationor position of the work tool 104 and/or the attachment coupler 106relative to the machine 102 and/or the machine component 108. Theelectronic controller unit 126 may be configured to detect the work tool104 and determine the occlusion of the work tool 104 and/or theattachment coupler 106 based, at least partially, upon the informationin the input signal 518. In another aspect, the electronic controllerunit 126 is coupled to the IMU 128 to receive an input signal 520 fromthe IMU 128. The input signal 520 may include data related to adead-reckoning of the machine 102.

In one aspect of this disclosure, the electronic controller unit 126includes the input-output port 504, a processor 506, and the memory 508coupled to each other, for example, by an internal bus (not shown). Theelectronic controller unit 126 may include additional components knownto one of ordinary skill in the art, which components are not explicitlyillustrated in FIG. 5. For example, the electronic controller unit 126may include a programmable logic circuit (PLC), a timer/clockingcircuit, heat sinks, visual indicators (e.g., light emitting diodes),impedance matching circuitry, internal buses, co-processors or monitorprocessors, batteries and power supply units, power controller chips,transceivers, wireless modules, satellite communication processingmodules, and embedded systems on various integrated chips. In oneaspect, the electronic controller unit 126 may be separate from anengine controller unit (not shown). In an alternative aspect, theelectronic controller unit 126 may be integrated with or may share spaceand processing resources with the engine controller unit.

The input-output port 504 may be a single port or a collection of ports.The input-output port 504 is configured to transmit and receive variousinputs and data from other parts of the machine 102 and forward suchinputs and data to the processor 506. In one aspect, the input-outputport 504 may be two separate ports, one configured to receive variousinput signals from various parts of the machine 102 (e.g., the sensor110, the IMU 128, etc.) and another configured to output signals fordisplay (e.g., on the output device 140) or for control of the machine102 (e.g., to the machine control system 130). Alternatively, thefunctionalities of inputting and outputting maybe integrated into asingle port illustrated as the input-output port 504 in FIG. 5.

In one aspect, the processor 506 is a hardware device such as anintegrated circuit (IC) chip fabricated to implement various featuresand functionalities of the aspects discussed herein. By way of exampleonly and not by way of limitation, the processor 506 may be fabricatedusing a Complementary Metal Oxide Semiconductor (CMOS) fabricationtechnology. In one aspect, the processor 506 may be implemented as anApplication Specific Integrated Circuit (ASIC), a Field ProgrammableGate Array (FPGA), a System-on-a-Chip (SOC), or the like. In anotheraspect, the processor 506 may include components such as packaging,input and output pins, heat sinks, signal conditioning circuitry, inputdevices, output devices, processor memory components, cooling systems,power systems and the like, which are not shown in FIG. 5. In oneaspect, the processor 506 is configured to execute various parts of amethod 800 illustrated in FIG. 8 by executing computer executableinstructions 510 in the memory 508. In yet another aspect, the processor506 may be a plurality of processors arranged, for example, as aprocessing array.

The memory 508 may be implemented as a non-transitory computer readablemedium. By way of example only, the memory 508 may be a semiconductorbased memory device including but not limited to random access memory(RAM), read only memory (ROM), Dynamic RAM, Programmable ROM,Electrically Erasable programmable ROM (EEPROM), Static RAM, Flashmemory, combinations thereof, or other types of memory devices known toone of ordinary skill in the art. In one aspect, the memory 508 iscoupled to the processor 506 directly via a communication and signalbus. In one aspect, the memory 508 may be made of or implemented using anon-transitory computer readable storage medium on which the computerexecutable instructions 510 reside. The computer executable instructions510 when executed by the processor 506 cause the processor 506 to carryout the features and functionalities of the various aspects of thisdisclosure, such as those discussed with respect to FIG. 8. Suchnon-transitory computer readable storage medium may includesemiconductor memory, optical memory, magnetic memory, mono- orbi-stable circuitry (flip-flops, etc.) and the like, or combinationsthereof. Such non-transitory computer readable storage medium excludessignals that are transitory.

The computer executable instructions 510 may be executed by theprocessor 506 using high-level or low-level compilers and programminglanguages (e.g., C++). In one aspect, the computer executableinstructions 510 may be executed remotely by a base station, and resultsof such execution provided to the processor 506 for controlling anoutput of the machine 102 to install the work tool 104 to the machine102. In this respect, it will be appreciated that the specific locationof the computer executable instructions 510 inside the memory 508 is byway of example only, and not by way of limitation.

In one aspect, the memory 508 includes or is coupled to a database 512.The database 512 includes images of a plurality of work tools, includingthe work tool 104. Such images are saved as a library of image files andcomputerized models in the database 512. Such models may include or maybe used to generate three-dimensional and two dimensional views of theplurality of work tools attachable to the machine 102, including thework tool 104. Each such image or model in the database 512 includes animage of the respective attachment couplers of the work tools. Forexample, an image of the work tool 104 in the database 512 includes animage of the attachment coupler 106. Such image files may be in standardformat (e.g., JPEG) known to those of ordinary skill in the art. Thedatabase 512 may further include various numerical parameters associatedwith one or more dimensions of the work tool 104 and the attachmentcoupler 106, as well as other identification information associated withthe work tool 104 and the attachment coupler 106. In one aspect, theprocessor 506 may be able to generate an image of the work tool 104 andthe attachment coupler 106 based upon the numerical parameters of thework tool 104 and the attachment coupler 106 stored in the database 512.Such images and information may be continuously accessible to theprocessor 506 before, during, and after an occlusion of the work tool104 and/or the attachment coupler 106 in the field of view 502 of thesensor 110 occurs.

Referring to FIG. 6, a frame of a three-dimensional scene 600 capturedby the sensor 110 and outputted by the processor 506 on the outputdevice 140 is illustrated, in accordance with an aspect of thisdisclosure. The three-dimensional scene 600 is a reconstruction of anactual work environment in which the machine 102 and the work tool 104are placed. The three-dimensional scene 600 is a real time snapshot ofthe work environment as viewed by the operator of the machine 102, forexample, via the sensor 110 or directly. For example, thethree-dimensional scene 600 may be associated with an initial relativeposition of the machine 102 with respect to the work tool 104, orvice-versa. In the three-dimensional scene 600, the machine 102 isrepresented as a machine image 602, the machine component 108 as amachine component image 608, the work tool 104 is represented as a worktool image 604, and the attachment coupler 106 is represented as anattachment coupler image 606 at a first instance in time. In FIG. 6, thesensor 110 has a clear view of the work tool 104 as illustrated by thework tool image 604, and the attachment coupler 106 as illustrated bythe attachment coupler image 606.

Referring to FIG. 7, another frame corresponding to a three-dimensionalscene 700 captured by the sensor 110 and outputted by the processor 506on the output device 140 is illustrated, in accordance with an aspect ofthis disclosure. The three-dimensional scene 700 is anotherreconstruction of the actual work environment in which the machine 102and the work tool 104 are placed at a second time instance subsequent tothe first time instance of FIG. 6. Similar to the three-dimensionalscene 600, the three-dimensional scene 700 is a real time snapshot ofthe work environment as viewed by the operator of the machine 102, forexample, via the sensor 110 or directly, at the second time instance. Inthe three-dimensional scene 700, the machine 102 is represented as amachine image 702 and the work tool 104 is represented as a work toolimage 704 at the second instance in time. However, due to an occlusioncaused by the machine component 108 represented as a machine componentimage 708, the attachment coupler 106 is not visible to the operator orthe sensor 110 in the three-dimensional scene 700. As a result, theattachment coupler image 606 previously shown on the output device 140is at least partially missing in FIG. 7.

It will be appreciated that the three-dimensional scene 600 and thethree-dimensional scene 700 are two visual examples of the machine 102in operation as outputted on the output device 140, but the outputdevice 140 may continuously display a plurality of three-dimensionalscenes on a frame-by-frame basis as provided by the processor 506 to theoutput device 140 based upon the input signals (including the inputsignal 518) from the sensor 110. In one aspect, the three-dimensionalscene 600 and the three-dimensional scene 700 may be provided on adisplay of a remote operator of the machine 102 in a remote base station(not shown) as a real-time video of the work scene in which the machine102 and the work tool 104 are deployed. Such frame-by-framerepresentation of the work environment of the machine 102 when used forrecognition and subsequent installation of the work tool 104 may be in asimulation format, or may be used as a simulation to train operators ofthe machine 102 to install the work tool 104.

INDUSTRIAL APPLICABILITY

The present disclosure is applicable to accurately recognizing worktools for installation to machines using computer-vision.

Machines have multiple types of work tools (attachments) for differentwork purposes. For example, a wheel loader may use a bucket for movingearth, and may use a fork for picking up pallets. Changing work tools ina safe and quick fashion is one of the basic qualifications of a goodmachine operator. However, it may require a new operator a long time oftraining to master this skill.

A typical challenge encountered by the machine operator during achanging of the work tool is that additional manual operations may berequired for changing the machine's output mode settings. Usually, themachine operator has to select a proper machine mode for a specific worktool by manually pressing a button on control panel. Forgetting to pressthe button or inadvertently selecting a wrong mode may cause the machineto malfunction.

Another challenge for the machine operator is a limited front field ofview of the work tool. The view of the work tool to be installed orattached may be blocked by one or more mechanical parts on the machine.For example, the hydraulic cylinders and mechanical linkages in front ofan operator cab of a wheel loader may block the machine operator's viewof a fork to be installed or attached to the machine.

Yet another challenge faced by the machine operator is the difficulty ofaligning an attachment coupler of a work tool at a distance from themachine to which the work tool attaches. When the work tool to beinstalled is at a distance from a machine (e.g., an excavator), it isdifficult for the machine operator to accurately align the work toolwith an attachment coupler due to the distance.

Various aspects of this disclosure address the complex problem ofrecognizing and installing or attaching the work tool 104 when there isan occlusion of the work tool 104 with respect to the field of view 502of the sensor 110, for example, due to the machine component 108 beingpresent between the sensor 110 and the attachment coupler 106 of thework tool 104.

Referring to FIG. 8, there is illustrated a method 800 for installingthe work tool 104 for the machine 102, in accordance with an aspect ofthis disclosure. FIG. 8 presents the method 800 as a flow diagram,although the method 800 may be understood using other types ofpresentations such as process diagrams, graphs, charts, equations, etc.In one aspect, one or more processes or operations in the method 800 maybe carried out by the electronic controller unit 126 inside the machine102. For example, the one or more processes or operations may be carriedout by the processor 506 inside the electronic controller unit 126,using various input signals from the sensor 110 and/or the IMU 128, andby executing the computer executable instructions 510 stored in thememory 508 of the electronic controller unit 126. As discussed, theinput signals from the sensor 110 and/or the IMU 128 may be received atthe electronic controller unit 126 and processed by the processor 506while the machine 102 is in use or is in operation in a work environmentcaptured, for example, as the three-dimensional scene 600.

In another aspect, in the method 800, one or more processes oroperations, or sub-processes thereof, may be skipped or combined as asingle process or operation, and a flow of processes or operations inthe method 800 may be in any order not limited by the specific orderillustrated in FIG. 8. For example, one or more processes or operationsmay be moved around in terms of their respective orders, or may becarried out in parallel. The term “flow” generally refers to a logicalprogression of operations in an exemplary manner carried out by theprocessor 506 and/or other components of the electronic controller unit126. However, such a flow is by way of example only and not by way oflimitation, as at a time, the flow may proceed along multiple operationsor processes of the method 800. Further, the method 800 may be carriedout by the electronic controller unit 126 for other types of work toolsattachable to the machine 102 and is not limited to the work tool 104.The method 800 may be implemented by the processor 506 in a high levelor a low level programming language (e.g., C++, assembly language, etc.)using logic circuitry inside the electronic controller unit 126 and byexecuting the computer executable instructions 510 in the memory 508.Furthermore, communication between the processor 506 and the database512 may occur using standard querying routines (e.g., Sequential QueryLanguage or SQL) and one or more search algorithms may be implementedusing the computer executable instructions 510 by the processor 506 tosearch the database 512. Such search algorithms may be in assemblylanguage or a high level programming language, as will be appreciated byone of ordinary skill in the art, in view of this disclosure.

The method 800 may begin in an operation 802 in which the processor 506is configured to capture a work scene in which the machine 102 and thework tool 104 are deployed. Such capturing of the work scene may be doneas a three-dimensional capture, as indicated, for example, in thethree-dimensional scene 600 and the three-dimensional scene 700. Thethree-dimensional capture of the work scene may be based upon continuousinputs from the sensor 110 and/or the IMU 128. For example, when thesensor 110 is a monocular or a stereo camera, the three-dimensionalscene 600 may be captured as an initial work scene. The sensor 110 maycontinue capturing the work scene on a frame-by-frame basis by capturingthe three-dimensional scene 700 and subsequent three-dimensional scenes.In one aspect, the three-dimensional scene 600 and the three-dimensionalscene 700 may be captured by the sensor 110 and outputted by theprocessor 506 on the output device 140 (as illustrated in FIGS. 6 and 7)as a time series of a plurality of three-dimensional scenes. In oneaspect, such three-dimensional scenes may be viewable as a video by anoperator of the machine 102.

In one aspect, the work scene captured by the sensor 110 may not have adesired level of detail. In this scenario, the processor 506 may addvarious additional features of the work scene based upon prior knowledgeof the work scene, using the database 512 to create thethree-dimensional scene 600. For example, when the sensor 110 is aninfrared camera operating at night, the processor 506 may create asimulation of the work scene to add more details (e.g., surroundingstructures, color, etc.) to the captured work scene. Such details maythen be presented as part of a simulation of the three-dimensional scene600 and/or the three-dimensional scene 700 on the output device 140. Theoperation 802 may be initiated manually (e.g., by an operator) orautomatically (e.g., every time the machine 102 is started).

In an operation 804, the processor 506 detects the work tool 104 in thecaptured work scene. The work scene may include a plurality of objectsaround the machine 102. The processor 506 may detect the work tool 104based upon the input signal 518 from the sensor 110. In one aspect, theinput signal 518 may be provided every 30 s to the processor 506,although a frequency at which the input signal 518 is generated by thesensor 110 may be programmable and variable. For example, the inputsignal 518 from the sensor 110 may include information related to aplurality of shapes detected by the sensor 110 that are in the field ofview 502 of the sensor 110. Upon receiving the input signal 518, theprocessor 506 may send a query to the database 512 to obtain one or moredimensions of the work tool 104 stored as a three-dimensional model ofthe work tool 104 in the database 512, as well as three-dimensionalmodels of other work tools and objects that may correspond to theplurality of shapes detected by the sensor 110. The processor 506 maydetect the work tool 104 by extracting visual and/or geometric featurescorresponding to computer vision feature descriptors. Such featuredescriptors may include, but are not limited to, a histogram of orientedgradients, speeded-up robust features, scale-invariant featuretransform, and the like, or combinations thereof.

In another aspect, the processor 506 may extract the visual/geometricfeatures of the objects from the input signal 518 and compare thesefeatures with the three-dimensional model library in the database 512.If there is a match, the processor 506 determines that the work tool 104has been detected.

In yet another aspect, as part of the detection of the work tool 104 bythe processor 506, the work tool 104 may reflect an outputelectromagnetic, acoustic, and/or optical signal from the sensor 110.Such a reflected signal when captured back by the sensor 110 is providedto the processor 506 as the input signal 518. The input signal 518 mayinclude optical intensity variations indicating a presence of the worktool 104. The processor 506 detects the work tool 104 based, forexample, on the optical intensity variations in the input signal 518. Incontrast, when the work tool 104 is not present or is not detected, theinput signal 518 from the sensor 110 may be missing or may not have anyinformation regarding the presence of the work tool 104. In one aspect,processor 506 may detect the work tool 104 based upon an input providedto the processor 506 by an operator of the machine 102. The operator mayview the work tool 104 directly. Alternatively, the operator may obtaininformation regarding a presence of the work tool 104 on the outputdevice 140. The processor 506 may apply image processing to detectimages captured by the sensor 110, e.g., when the sensor 110 is anoptical camera.

In an operation 806, the processor 506 carries out classifying the worktool 104 detected in the operation 804. The processor 506 obtains animage of the work tool 104 based upon the input signal 518 from thesensor 110. The processor 506 may then extract data associated with thephysical features of the work tool 104. Such data may include, forexample, dimensions and shape of the work tool 104, determined, forexample, in a manner similar to the extraction of features of the worktool 104 for the detection performed in the operation 804. The processor506 may communicate with the database 512 to obtain a match of the worktool 104 with a known work tool whose information is stored in thedatabase 512. For example, the database 512 may store a library ofcomputerized models of a plurality of types of work tools that may beattached to the machine 102, and one of such computerized models maymatch the work tool 104, based upon the information in the input signal518.

In one aspect, a search algorithm may be executed by the processor 506to search for data matching the data associated with the work tool 104,as detected. For example, once a height, length, and/or depth of thework tool 104 have been determined by the processor 506 based uponextracting such information from the input signal 518 of the sensor 110,the processor 506 may send a query to the database 512, to determinewhether the database 512 has matching work tools meeting satisfying thecriteria corresponding to the dimensions of the work tool 104 in thequery. The database 512 may communicate back one or more matches of thework tool 104. When there is more than one match, the processor 506 mayapply additional criteria to identify and accurately recognize the worktool 104. For example, the database 512 may indicate to the processor506 that there are two work tools matching the dimensions of the worktool 104.

However, the processor 506 may determine that only one of the twomatching work tools are appropriate for the machine 102 and/or the workenvironment in which the machine 102 is deployed. Accordingly, theprocessor 506 classifies the work tool 104 by selecting informationregarding the correctly matched known work tool provided by the database512, and rejects the other matching results from the database 512. Inthis respect, the work tool 104 is identified or recognized usingcomputer vision. The term “computer vision” may relate to processinginformation related to the work tool 104 to determine a type,dimensions, and a positioning of the work tool 104 relative to themachine 102, and further, applying the processed information to assistinstallation of the work-tool using generated images of the work tool104 (e.g., the work tool image 604 in FIG. 6).

In an operation 808, the processor 506 may select an output mode of themachine 102 based upon the classifying carried out in the operation 806.The output mode for the machine 102 may be set automatically based uponthe classifying of the work tool 104. Alternatively, the processor 506may output an indication to the output device 140 based upon which anoperator can select the output mode of the machine 102. In one aspect,when the processor 506 automatically sets the output mode of the machine102 based upon the classifying, manual errors due to inadvertenterroneous selection by the operator of the machine 102 may be avoided.The output mode selection determines an amount of power to betransmitted to the work tool 104, for example, by the machine component108 to which the work tool 104 is attached. Such an output power iswithin a range to avoid potentially improper functioning or potentialdamage to the work tool 104 and/or to the work surface upon which thework tool 104 operates. For example, based upon the output mode, theprocessor 506 may output signals to the machine control system 130 tooutput appropriate electrical current to one or more motors or hydraulicactuators of the machine 102, which motors or hydraulic actuators maythen control a motion of the machine 102 and/or the machine component108.

In an operation 810, once the work tool 104 has been detected by theprocessor 506, the processor 506 may determine the firstthree-dimensional location 120 of the work tool 104. The firstthree-dimensional location 120 may be an initial location or position ofthe work tool 104 from the machine 102, for example, when the work toolinstallation system 100 is initiated. Alternatively, the firstthree-dimensional location 120 may refer to any initial location orposition of the work tool 104 with respect to a calculation cycle ortime slot of the processor 506 for which the method 800 is beingimplemented or carried out. For example, the processor 506 may need toupdate a previous determination of a calculated three-dimensionalposition of the work tool 104. To do so, the processor 506 may restartcalculating and may use the first three-dimensional location 120 as astarting point or starting value to determine subsequentthree-dimensional locations of the work tool 104 (e.g., the secondthree-dimensional location 220).

In one aspect, the processor 506 determines the first three-dimensionallocation 120 based upon a knowledge of a position of the sensor 110 anddetermining a propagation time for a signal emitted by the sensor 110towards the work tool 104 to be reflected back and received at thesensor 110. Upon reception of the reflected signal, the sensor 110provides the input signal 518 to the processor 506. For example, theprocessor 506 may utilize such a determination of the firstthree-dimensional location 120 when the sensor 110 is an activeperception sensor, such as a Lidar, a time-of-flight three-dimensionalcamera, radar, and the like, or combinations thereof.

However, when the sensor 110 is a passive perception sensor, such as amonocular color camera, a monocular infrared camera, or a binocularstereo camera, the processor 506 may apply computer vision algorithms todetermine the first three-dimensional location 120. For monocularcameras used as the sensor 110, the processor 506 may use computervision technologies such as “structure-from-motion” to generate, forexample, the three-dimensional scene 600 in the field of view 502 of thesensor 110. For binocular stereo cameras used as the sensor 110, theprocessor 506 may use a stereo triangulation to generate, for example,the three-dimensional scene 600 in the field of view 502 of the sensor110. Once the three-dimensional scene 600 has been reconstructed by theprocessor 506, the first three-dimensional location 120 of the work tool104 is calculated with respect to the sensor 110. Since the sensor 110location with respect to the machine 102 are known (e.g., (x, y, z)coordinates, roll, pitch, and yaw), the first three-dimensional location120 of the work tool 104 with respect to the machine 102 may beobtained. It will be appreciated by one of ordinary skill reading thisdisclosure that the second three-dimensional location 220, the thirdthree-dimensional location 122, and the fourth three-dimensionallocation 222, or any other three-dimensional locations in the work toolinstallation system 100, may be determined by the processor 506 in amanner similar to the determination of the first three-dimensionallocation 120 by the processor 506.

In one aspect, the processor 506 may use a linear relationship todetermine the first three-dimensional location 120 of the work tool 104.By way of example only and not by way of limitation, the processor 506may know a three-dimensional position of the sensor 110 on the machine102 in Cartesian coordinates (e.g., X, Y, Z values) and may obtain thisinformation from the memory 508. Further, the processor 506 may obtainphysical dimensions of the machine 102 and the machine component 108 anddetermine the relative position of the work tool 104 with respect to themachine 102 and the machine component 108.

In an operation 812, the processor 506 may determine the thirdthree-dimensional location 122 of the attachment coupler 106 of the worktool 104. In one aspect, the third three-dimensional location 122 isdetermined based upon the classifying of the work tool 104 carried outin the operation 806. Upon classifying, the processor 506 may know wherethe attachment coupler 106 is located on the work tool 104 based uponthe data for the work tool 104 retrieved from the database 512. In thisrespect, the processor 506 does not have to rely upon explicit inputsignals from the sensor 110 regarding the third three-dimensionallocation 122 of the attachment coupler 106. Rather, once the work tool104 has been recognized, based upon the classifying, the processor 506uses the information regarding the work tool 104 from the database 512to determine the third three-dimensional location 122 of the attachmentcoupler 106.

In one aspect, such a determination of the relative position of theattachment coupler 106 is part of the computer vision baseddetermination utilized by the processor 506. Further, such computervision assisted determination of the relative position of the attachmentcoupler 106 may include determining a precise position of the attachmentcoupler 106 on the work tool 104 using the information obtained from thedatabase 512 regarding the work tool 104. For example, when the worktool 104 is a bucket, an attachment pin of the bucket and a location ofthe attachment pin may be determined by the processor 506 upon detectionand classification of the work tool 104 (e.g., as carried out in theoperations 804 and 806). Accordingly, the processor 506 may form acomplete three-dimensional positional view of the work tool 104independent of an actual physical view of the work tool 104. Thethree-dimensional view of the work tool 104 may then be provided to theoutput device 140 as an output by the processor 506. Such an output maybe represented as part of the three-dimensional scene 600 and thethree-dimensional scene 700 illustrated in FIGS. 6 and 7, respectively.

In an operation 814, the processor 506 may continuously track the worktool 104 and the attachment coupler 106 of the work tool 104 using theinput signal 518 from the sensor 110. The tracking of the work tool 104and/or the attachment coupler 106 may be based upon determining arelative distance between the machine 102 and the attachment coupler 106starting, for example, with the third three-dimensional location 122 ofthe attachment coupler 106. As the machine 102 moves toward the worktool 104, the processor 506 applies computer-vision based on imageinformation about the work tool 104 and the attachment coupler 106 fromthe database 512 to accurately track the work tool 104 and/or theattachment coupler 106 to adjust and control the motion of the machine102.

In addition to the input signal 518, the processor 506 is configured forthe tracking based on the classification of the work tool 104 carriedout in the operation 806. Since the processor 506 knows the exact typeof the work tool 104, the processor 506 can determine the type andlocation of the attachment coupler 106 on the work tool 104. Then, basedupon a velocity of the machine 102, and hence the sensor 110, theprocessor 506 can continuously track where the attachment coupler 106 onthe work tool 104 is located. The tracking of the work tool 104 and/orthe attachment coupler 106 may be viewable on the output device 140. Forexample, the work tool 104 and/or the attachment coupler 106 as trackedmay be presented on the three-dimensional scene 600 for an operator ofthe machine 102. Such tracking of the work tool 104 and the attachmentcoupler 106 may be referred to as “visual odometry” in which a velocityof the machine 102 is used in conjunction with the images of the worktool 104 retrieved from the database 512 by the processor 506 todetermine an accurate three-dimensional localization of the attachmentcoupler 106. Such three-dimensional localization of the attachmentcoupler 106 may be indicated on the output device 140, for example, asgeographical coordinates and directions.

In one aspect, such tracking may be carried out on a frame-by-framebasis based on input signals (including the input signal 518) from thesensor 110 received at the processor 506. For example, thethree-dimensional scene 600 may be considered as a first frame on theoutput device 140 in which the work tool 104 and/or the attachmentcoupler 106 are presented and tracked. Likewise, subsequentthree-dimensional scenes (e.g., the three-dimensional scene 700) may bepresented as a time based progression of the captured work scene. Ineach such frame, the work tool 104 and/or the attachment coupler 106 aretracked as indicated by the relative positions of the work tool 104and/or the attachment coupler 106 with respect to the machine 102. Itwill be appreciated that the tracking of the work tool 104 and/or theattachment coupler 106 on a frame-by-frame basis may be for visualpresentation of the simulated locations of the work tool 104 and/or theattachment coupler 106 (directly correlated with the actual physicallocations of the work tool 104 and/or the attachment coupler 106).However, for purposes of calculating the relative positions of themachine 102 and the work tool 104 and/or the attachment coupler 106(e.g., using the first three-dimensional location 120, etc.), such avisual presentation is an example only, and not a limitation. Forexample, the processor 506 may determine the relative positions based onnumerical or tabular values that are updated as the work tool 104 and/orthe attachment coupler 106 are tracked.

In an operation 816, the processor 506 may detect an occlusion of thework tool 104. Such occlusion may be partial (e.g., as shown in FIG. 4)or may be complete. Further, the occlusion of the work tool 104 mayrefer to complete or partial blocking of a view of the attachmentcoupler 106. Such occlusion may be viewable on the output device 140 asthe three-dimensional scene 700 in which the attachment coupler image606 from FIG. 6 is unavailable, or partially unavailable, to theoperator of the machine 102. By way of example only and not by way oflimitation, the occlusion of the work tool 104 may occur due to apresence of the machine component 108 in the field of view 502 of thesensor 110 and/or the view of the operator of the machine 102. It willbe appreciated that such occlusion may occur due to a variety of reasonsin addition to or other than the presence of the machine component 108.For example, the occlusion may occur due to poor weather conditions, orpresence of other physical objects blocking the view of the work tool104 and/or the attachment coupler 106, or combinations thereof. In thisrespect, the term “occlusion” may refer to a lack of availability of aview of the work tool 104 in general, and of the attachment coupler 106,in particular, for any reason. Such lack of availability of a clear viewof the attachment coupler 106 may make it difficult and time consumingto manually install the work tool 104 to the machine component 108.

In one aspect, the occlusion may be detected by the processor 506 basedupon the continuous tracking performed in the operation 814. Forexample, the processor 506 may provide the three-dimensional scene 600as a first frame on the output device 140. In the first frame on thethree-dimensional scene 600, the work tool 104 and the attachmentcoupler 106 are clearly visible correspondingly as the work tool image604 and the attachment coupler image 606. The processor 506 may continueto provide a plurality of such frames as the machine 102 moves towardsthe work tool 104 to install the work tool 104.

However, in one such plurality of frames represented, for example, asthe second frame in the three-dimensional scene 700, the attachmentcoupler image 606 may not be available and may be partially or fullyblocked. In this second frame, the processor 506 detects that the worktool 104 has been occluded since the attachment coupler image 606 isdetected to be partially or fully missing. The processor 506 carries outthe occlusion detection using the input signal 518 that informs theprocessor 506 about the objects in the work scene in which the machine102 is deployed. As indicated by information in the input signal 518 tothe processor 506, various machine components, including the machinecomponent 108, are detected in three-dimensional space around themachine 102. Therefore, the processor 506 knows if one or more of themachine components (e.g., the machine component 108) come into the lineof sight between the sensor 110 and the work tool 104, which causes fullocclusion or partial occlusion. The processor 506 keeps tracking apreviously detected object (e.g., the work tool 104) on a frame-by-framebasis. In the meanwhile, the processor 506 keeps performing objectdetection (e.g., in the operation 804) and classification on each frame(e.g., in the operation 806). If the processor 506 finds that the worktool 104, as tracked, shows different geometric features between twoadjacent frames (e.g., between the first three-dimensional scene 600 andthe second three-dimensional scene 700), the processor 506 determinesthat an occlusion of the work tool 104 has happened.

In another aspect, the processor 506 may determine, based upon thevisual odometry performed in the operation 814 that the machinecomponent 108 is moving with a known velocity towards the work tool 104and/or the attachment coupler 106. Based upon the firstthree-dimensional location 120, the second-three-dimensional location220 and/or the third three-dimensional location 122 determined in theoperations 810 and 812, as well as the type of the work tool 104classified in the operation 806, the processor 506 may calculate anexact location of the work tool 104 and the attachment coupler 106.Accordingly, even during the occlusion detected by the processor 506,the method 800 for installing the work tool 104 to the machine 102 doesnot stop and the machine 102 continues moving towards the work tool 104and the attachment coupler 106 to install the work tool 104 to themachine 102. In yet another aspect, the processor 506 may seekconfirmation regarding the occlusion from the operator of the machine102, or may receive an input from a human operator of the machine 102that an occlusion has been visually detected, in addition to orindependently of the detection of occlusion determined by the processor506 itself in the operation 816.

In an operation 818, the processor 506 may update the three-dimensionalscene 600 based on the detected occlusion in the operation 816. Thethree-dimensional scene 600, as updated to reflect the detectedocclusion of the work tool 104, may be provided as the three-dimensionalscene 700 outputted on the output device 140. The processor 506 mayfurther carry out the updating of the three-dimensional scene 600 basedupon information captured by the sensor 110 in the operation 802initially. For example, the sensor 110 may include information regardingstatic structures (trees, buildings, etc.) in or around the work sceneof the machine 102. The processor 506 may use such static structure, inaddition to the dynamically changing positions of the various objects inthe field of view 502, to update the three-dimensional scene 600.

In one aspect, the processor 506 may update the three-dimensional scene600 using the tracking carried out in the operation 814. For example,the three-dimensional scene 700 may show an updated or current relativeposition of the work tool 104 and/or the attachment coupler 106 withrespect to the machine 102 and/or the machine component 108 based uponthe tracking by the processor 506. In another example, even after theocclusion has been detected, the processor 506 may continue updating thethree-dimensional scene 600 using computer vision since the processor506 has knowledge of the physical features of the work tool 104 and alast known location of the work tool 104 and/or the attachment coupler106, as well as the velocity of the machine 102, among other parameters(e.g., additional inputs from the sensor 110, the IMU 128, GPS locationof the machine 102, etc.).

In an operation 820, the processor 506 may determine a dead-reckoning ordeduced reckoning of the machine 102. The processor 506 may receive asecond input signal (e.g., the input signal 520) from the IMU 128coupled to the electronic controller unit 126. The second input signalis used by the processor 506 to determine a motion of the machine 102 inthree-dimensional space. As known, the IMU 128 may receive inputs fromvarious inertial sensors (not shown) attached to the machine 102 togenerate the second input signal to the processor 506. In one aspect,the processor 506 may further utilize a satellite signal or a signalfrom an unmanned aerial vehicle to corroborate the information in thesecond input signal regarding the dead-reckoning of the machine 102. Inanother aspect, the IMU 128 may be optional for purposes of thisdisclosure. However, using the second input signal from the IMU 128, inaddition to the visual odometry and computer vision based trackingperformed by the processor 506, may result in a more robust estimate ofthe first three-dimensional location 120, the second three-dimensionallocation 220, the third three-dimensional location 122, and the fourththree-dimensional location 222 of the work tool 104 and the attachmentcoupler 106, respectively, with respect to the sensor 110 on the machine102.

In an operation 822, the processor 506 may perform a fusion of the firstinput signal (e.g., the input signal 518) from the sensor 110 and thesecond input signal (e.g., the input signal 520) from the IMU 128 togenerate a combined signal that provides a current or most recentrelative positioning between the machine 102 and the work tool 104. Thecombined signal may be generated by the processor 506 by normalizing thefirst input signal and the second input signal to a common format, andthen verifying whether the information regarding the motion of themachine 102 as deduced from the visual odometry and the dead-reckoningare same or in a similar range of values.

In an operation 824, based upon the fusion of the first and the secondinput signals from the sensor 110 and the IMU 128, respectively, theprocessor 506 determines an updated position of the work tool 104relative to the machine 102. For example, the updated position of thework tool 104 may be obtained by calculating the secondthree-dimensional location 220, after the occlusion has been detected inthe operation 816.

Likewise, in an operation 826, the processor 506 determines the fourththree-dimensional location 222 of the attachment coupler 106 based uponthe combined signal obtained in the operation 822. For example, once anupdated position of the work tool 104 is determined by the processor506, based upon one or more of the classification of the work tool 104,the dead-reckoning on the machine 102, and the third three-dimensionallocation 122 of the attachment coupler 106, the processor 506 maydetermine the fourth three-dimensional location 222 even after theocclusion occurs.

In an operation 828, the processor 506 may output an updated position ofthe work tool 104 and the attachment coupler 106 based upon theoperations 824 and 826 to the machine control system 130. For example,the processor 506 may output the second three-dimensional location 220of the work tool 104 and the fourth three-dimensional location 222 ofthe attachment coupler 106 as part of the updated position of the worktool 104 and the attachment coupler 106. In one aspect, the processor506 may output the updated position of the work tool 104 and/or theattachment coupler 106 in the three-dimensional scene 700 to theoperator of the machine 102. Such outputting of the updated positions ofthe work tool 104 and/or the attachment coupler 106 may be provided on acontinuous time basis as the machine 102 moves and the occlusion of thework tool 104 and/or the attachment coupler 106 changes, or evenvanishes after a certain time has passed without stopping the machine102 or without disrupting the process of installing the work tool 104 tothe machine 102.

In an operation 830, the processor 506 may control the machine 102and/or the machine component 108 based upon the updated positions of thework tool 104 and the attachment coupler 106 using the machine controlsystem 130. Based upon the updated positions, the machine control system130 may provide a response signal to the processor 506 to control amotion of the machine 102 and/or the machine component 108. For example,the updated position may require the machine 102 to move in a particularspatial direction at a speed different from a current speed of themachine 102 upon detecting the occlusion in the operation 816.Similarly, the updated position of the work tool 104 and the attachmentcoupler 106 may require the machine component 108 to move in a specificdirection and speed different from a current direction in which themachine component 108 was moving. For example, the machine component 108might have been moving at a particular angular acceleration initially,prior to the occlusion. The machine control system 130 may receive anoutput signal from the processor 506 to reduce the angular accelerationwith which the machine component 108 may be rotating upon detection ofthe occlusion. As a result, the machine component 108 may be controlledto move in a more accurate manner even though the attachment coupler 106is partially of fully occluded in the field of view 502 of the sensor110. It will be appreciated that the processor 506 may control themotion of the machine 102 and/or the machine component 108 in otherways, for example, along linear and/or circular directions, dependingupon the type of the work tool 104 and/or the attachment coupler 106.

In an operation 832, the processor 506 determines that the machinecomponent 108 and the attachment coupler 106 are at an appropriatedistance and relative orientation for the attachment coupler 106 tocouple to or attach to the machine component 108. The processor 506 thenoutputs signals to the machine control system 130 to attach the worktool 104 to the machine component 108 of the machine 102 at theattachment coupler 106. Based upon the classifying of the work tool 104and the attachment coupler 106 (in operation 806), and the updatedpositions of the work tool 104 and the attachment coupler 106 (inoperations 824 and 826, respectively), the processor 506 knows thedimensions and the location of the attachment coupler 106 and installsthe attachment coupler 106, even when the attachment coupler 106 isoccluded in the field of view 502 of the sensor 110. Such installationmay be performed by the processor 506 outputting signals to the machinecontrol system 130 to open mechanical linkages on the machine component108 to couple to the attachment coupler 106.

In an operation 834, once the work tool 104 has been installed to themachine component 108, the processor 506 may output an indication of theinstallation of the work tool 104 to the operator of the machine 102.Such an indication may be on the output device 140, for example, as avisual, an audio, or an audio-visual indication inside the operator cab112 or to the remote base station, or both.

In an operation 836, upon receiving the indication from the processor506 that the work tool 104 has been installed to the machine 102, theoperator may operate the machine 102, as appropriate for the work sitein which the machine 102 is deployed. Such operating of the machine 102may further include outputting power from the machine 102 to the worktool 104 based upon the output mode selected in the operation 808specific to the type of the work tool 104 as determined in the operation806.

Various aspects of this disclosure aid the operator of the machine 102in the process of changing and attaching the work tool 104 when the worktool 104 and/or the attachment coupler 106 may be temporarily orotherwise be out of view (or, occluded) from the field of view 502 ofthe sensor 110. For example, when the sensor 110 is a camera, the worktool 104 may be out of the camera's view due to the motion of themachine 102 and/or the machine component 108. This may occur, forexample, when the camera is mounted on a top of the operator cab 112 andthe work tool 104 gets close to the machine 102 at an end of a work toolchanging and installation process. In one example, the hydrauliccylinders in the machine component 108 in front of the operator cab 112may partially block the field of view 502, because of which an operatorof the machine 102 may not be able to determine the firstthree-dimensional location 120 or the second three-dimensional location220 of the work tool 104.

To solve this complex problem, the processor 506 uses motion tracking tocontinuously estimate the location of the machine 102 inthree-dimensional space. Upon detection of an occlusion of the work tool104, the processor 506 can still calculate an updated position of thework tool 104 using a last seen location of the work tool 104 and theestimated motion of the machine 102 (received, for example, from the IMU128). Similar to a determination of the relative position of the worktool 104, the processor 506 determines the relative position of theattachment coupler 106 (e.g., an attachment pin) with respect to themachine 102 and/or the machine component 108 even after the occlusionhappens. As a result, various aspects of this disclosure ensure that theocclusion does not disrupt the process of installing the work tool 104to the machine 102 at the attachment coupler 106.

It will be appreciated that the foregoing description provides examplesof the disclosed system and technique. However, it is contemplated thatother implementations of the disclosure may differ in detail from theforegoing examples. All references to the disclosure or examples thereofare intended to reference the particular example being discussed at thatpoint and are not intended to imply any limitation as to the scope ofthe disclosure more generally. All language of distinction anddisparagement with respect to certain features is intended to indicate alack of preference for those features, but not to exclude such from thescope of the disclosure entirely unless otherwise indicated.

Recitation of ranges of values herein are merely intended to serve as ashorthand method of referring individually to each separate valuefalling within the range, unless otherwise indicated herein, and eachseparate value is incorporated into the specification as if it wereindividually recited herein. All methods described herein can beperformed in any suitable order unless otherwise indicated herein orotherwise clearly contradicted by context.

We claim:
 1. A method for installing a work tool for a machine,comprising: detecting, at an electronic controller unit of a machine, awork tool based upon a first input signal from a sensor coupled to theelectronic controller unit; determining, at the electronic controllerunit, a first three-dimensional location of the work tool relative tothe machine; detecting, at the electronic controller unit, an occlusionof the work tool; determining, at the electronic controller unit, asecond three-dimensional location of the work tool upon the detecting ofthe occlusion based on the first three-dimensional location; andcontrolling, at the electronic controller unit, a motion of the machinefor installing the work tool based upon the second three-dimensionallocation.
 2. The method of claim 1, further comprising: classifying, atthe electronic controller unit, the detected work tool based upon amatch with a known work tool in a database; and selecting automatically,at the electronic controller unit, an output mode of the machine basedupon the classifying.
 3. The method of claim 2, wherein the controllingthe motion of the machine is performed based upon the selecting of theoutput mode.
 4. The method of claim 2, further comprising: determining,at the electronic controller unit, a relative position of the machinewith respect to an attachment coupler of the work tool using a thirdthree-dimensional location of the attachment coupler, wherein thecontrolling is performed based upon the determining.
 5. The method ofclaim 1, further comprising: tracking, at the electronic controllerunit, the work tool based on the first input signal from the sensor; andoutputting, at an output device coupled to the electronic controllerunit, a three-dimensional scene of the tracked work tool based upon atleast one of the first three-dimensional location or the secondthree-dimensional location of the work tool, wherein the outputtingincludes displaying the occlusion of the work tool.
 6. The method ofclaim 5, wherein the tracking includes tracking a linear motion, anangular motion, or both, of the machine or a machine component.
 7. Themethod of claim 1, further comprising: determining, at the electroniccontroller unit, a dead-reckoning of the machine based upon a secondinput signal from an inertial measurement unit coupled to the electroniccontroller unit; and combining, at the electronic controller unit, thefirst input signal and the second input signal to update the secondthree-dimensional location of the work tool upon the detecting of theocclusion.
 8. The method of claim 1, further comprising: outputtingcontinuously, at the electronic controller unit, an updated position ofthe work tool and an attachment coupler of the work tool to a machinecontrol system; installing, using the electronic controller unit, thework tool to the machine at the attachment coupler based on the updatedposition; and indicating, from the electronic controller unit, to themachine control system that the installing is complete.
 9. The method ofclaim 1, wherein the detecting the occlusion of the work tool includesdetecting a machine component between the sensor and the work tool, themachine component blocking a field of view of the sensor.
 10. A worktool installation system, comprising: a machine attachable to a worktool, the machine including: a sensor; and an electronic controller unitcoupled to the sensor, the electronic controller unit configured to:detect the work tool based upon a first input signal from the sensor,determine a first three-dimensional location of the work tool relativeto the machine, detect an occlusion of the work tool, determine a secondthree-dimensional location of the work tool, when the occlusion isdetected, based on the first three-dimensional location, and control amotion of the machine for installing the work tool based upon the secondthree-dimensional location.
 11. The work tool installation system ofclaim 10, wherein the electronic controller unit is further configuredto: classify the detected work tool based upon a match with a known worktool in a database, and select an output mode of the machine based uponthe match.
 12. The work tool installation system of claim 11, whereinthe electronic controller unit is further configured to control themotion of the machine based upon the output mode.
 13. The work toolinstallation system of claim 11, wherein the electronic controller unitis further configured to: determine a relative position of the machinewith respect to an attachment coupler of the work tool using a thirdthree-dimensional location of the attachment coupler, wherein theelectronic controller unit is configured to control the motion of themachine based upon the determined relative position.
 14. The work toolinstallation system of claim 10, wherein the electronic controller unitis further configured to: track the work tool based on the first inputsignal from the sensor, and output, at an output device controlled bythe electronic controller unit, a three-dimensional scene of the trackedwork tool based upon at least one of the first three-dimensionallocation or the second three-dimensional location of the work tool,wherein the outputted three-dimensional scene includes the occlusion ofthe work tool.
 15. The work tool installation system of claim 14,wherein the electronic controller unit is configured to track a linearmotion, an angular motion, or both, of the machine or a machinecomponent.
 16. The work tool installation system of claim 10, furthercomprising: an inertial measurement unit on the machine coupled to theelectronic controller unit, wherein the electronic controller unit isfurther configured to: determine a dead-reckoning of the machine basedupon a second input signal from the inertial measurement unit, andcombine the first input signal and the second input signal to update thesecond three-dimensional location of the work tool when the occlusion isdetected.
 17. The work tool installation system of claim 10, wherein theelectronic controller unit is further configured to: output continuouslyan updated position of the work tool and an attachment coupler of thework tool to a machine control system; install the work tool to themachine at the attachment coupler based on the updated position; andindicate to the machine control system that the work tool is installed.18. The work tool installation system of claim 10, wherein the occlusionis detected by detecting a machine component between the sensor and thework tool, the machine component blocking a field of view of the sensor.19. The work tool installation system of claim 10, wherein the sensor isat least one camera on the machine.
 20. An electronic controller unit ofa machine comprising: a memory including computer executableinstructions for recognizing and installing a work tool to a machine;and a processor coupled to the memory and configured to execute thecomputer executable instructions, the computer executable instructionswhen executed by the processor cause the processor to: detect the worktool based upon a first input signal from a sensor, determine a firstthree-dimensional location of the work tool relative to the machine,detect an occlusion of the work tool, determine a secondthree-dimensional location of the work tool, when the occlusion isdetected, based on the first three-dimensional location, and control amotion of the machine for installing the work tool based upon the secondthree-dimensional location.